---
title: "Long-Term Effects & Universal Holdouts: When Your Lift Disappears"
description: "Short-run A/B tests systematically overstate long-run business impact due to novelty and learning effects. This post explores why your initial lift disappears over time and details the universal holdout designs built to catch it, ensuring you measure the true, sustained impact of your product changes."
date: 2026-10-05
categories: [A/B testing, experimentation, causal inference, product analytics, holdout groups]
#draft: true
---
## When your lift disappears
The product team at a consumer app ships a new notification system. A two-week A/B test comes in at +5% Daily Active Users, tight confidence interval, no SRM, no obvious spillover, CUPED-adjusted. The PM adds the win to the roadmap deck and moves on. Three months later, at the quarterly business review, aggregate DAU is flat. Not down, not up, flat. The same PM is now trying to explain, in front of the CFO, why the sum of six positive experiments this quarter has produced exactly zero movement in the top-line metric. The VP of Data is more uncomfortable still, because she has been running the numbers all week: each of the last six experiments was correctly powered, the p-values were clean, the CUPED-adjusted estimates were tight, and yet the portfolio is flat. Nothing was done wrong on any individual test. Everything is wrong at the portfolio level.
This is not a statistical anomaly and it is not a story about a rogue experiment. It is a structural property of how short-run experiments measure effects. Two weeks of exposure at 50/50 traffic is not a small-sample estimate of the long-run business impact of a permanent launch. It is a fundamentally different quantity, and treating it as an unbiased estimate of steady-state impact is the single most common measurement mistake I see in mature experimentation programs. The point is not that A/B tests are broken. The point is that A/B tests answer a narrower question than the one executives usually think they are answering, and the gap between the two is where roadmap credibility gets lost. The rest of this post is about why that gap exists, how to bound it, and what to instrument in its place.
Three mechanisms drive the wedge between the short-run experimental lift and the long-run business impact of a single feature. All three are documented in Kohavi, Tang and Xu (2020, Ch. 23), and each will get a full section later. Here I want to name them precisely so we share a vocabulary for the rest of the post.
The first is the novelty effect. Users have a finite attention budget, and anything new commands a temporary share of it. A new notification, a redesigned feed card, or a fresh banner gets clicked because it is unfamiliar, not because it is valuable. That share decays as the feature becomes part of the background, and a two-week window captures the peak of the decay curve rather than the steady state. Hohnhold, O'Brien and Tang (2015) estimate a learning half-life of roughly 60 days on Google search ads, which means that even a well-designed 90-day post-period measurement captures only about 65% of the true long-run learned effect, and a 14-day post-period captures closer to 60%. Translated to the notification example, a short-run +5% DAU can plausibly correspond to a steady-state +1% or less once novelty is amortised.
The second is the primacy effect, which is the mirror image. When the change imposes a friction cost, users initially resist it. They cannot find the button, the mental model is wrong, the muscle memory pulls them to the old flow. The two-week test then underestimates the true long-run benefit. Primacy dominates in complex UI redesigns and workflows where the user has invested in a habit, and it is why some launches that "failed" in test end up positive six weeks after full rollout. Novelty and primacy are the same phenomenon (user learning) with opposite sign, and Kohavi, Tang and Xu (2020, Ch. 23) treat them under a single framework.
The third is equilibrium effects, and this is the mechanism most likely to matter for the reader working at Bolt, Glovo, Uber, or Booking. As covered in [*When A/B Tests Lie*](https://joaovieira22.github.io/posts/AB%20testing%20com%20Spillovers/when_ab_tests_lie.html), spillovers violate SUTVA at the user level. Equilibrium effects are the aggregated, system-level version of the same problem. Consider a feature that improves the driver-side app in a ride-hailing marketplace, causing treated drivers to reposition more efficiently and complete an extra trip per shift. In a two-week user-level A/B test, treated drivers show +8% earnings and the launch looks obvious. But once the feature ships to 100% of drivers, aggregate supply in the busy zones rises, average rider ETAs fall by roughly 30 seconds, surge triggers less often, and per-trip revenue per driver drifts back toward the control mean. The user-level test measured a partial equilibrium quantity. The business runs in general equilibrium (Chamandy 2016; Kohavi, Tang and Xu 2020, Ch. 22).
Now stack this on top of the Winner's Curse: individual test estimates are upward-biased conditional on being shipped, because we select on statistical significance. Even if we corrected for the Winner's Curse and every shipped experiment were unbiased at its own decision boundary, the sum of ten unbiased +2% lifts would still not produce +20% at the aggregate level. Features interact. A new notification and a new email cadence compete for the same user attention. A homepage tile and a push notification cannibalise the same conversion. A retention feature raises the denominator against which a monetisation feature is measured. This is the fourth problem, the portfolio-level Winner's Curse, and it is not reducible to shrinkage on individual estimates. It is a portfolio-level phenomenon, and it is the subject of Section III.
The framing here matters, because it is easy to read what follows as an attack on A/B testing itself. It is not. A well-run A/B test answers a specific question well: what is the short-run causal effect of a single treatment on the users randomised into it. The problem this post is about is what happens when that answer is treated as if it were a different one, the cumulative long-run business impact of an entire roadmap on the whole user base. Those are two different quantities, and most of the QBR arguments I have seen between analytics and product leadership are about the second one while the dashboard is quietly reporting the first.
Which brings us to the question the rest of this post exists to answer. If short-run A/B tests are systematically upward-biased estimators of long-run business impact, and if the bias compounds across a shipping portfolio, what is the correct measurement tool at the business level? The answer, built up across Sections IV through VII, is the universal holdout: a small, permanently withheld control group that never receives any of the features shipped in a given period, and against which the full stack of launches is measured. It is the most practical instrument I know of for producing a defensible read on cumulative long-run business impact, and it is what separates experimentation programs that can defend their contribution at the quarterly review from those that cannot. It is also, as Section VI will make plain, expensive and easy to get wrong; nothing in this post should be read as a recommendation to run one without first understanding the invoice attached.
Before we can design that instrument, we need to understand each mechanism precisely, what drives it, how large it typically is in the wild, and what signals in the data reveal its presence. That is the job of Section II.
::: {.callout-tip title="Four mechanisms, four signatures"}
**Novelty effect.** Users engage more with anything new, because unfamiliarity commands a temporary share of their finite attention budget. That share decays as the feature becomes part of the background, with a half-life of weeks on typical Google-scale ad experiments. A two-week test captures peak novelty, not steady state, and overestimates the long-run effect (Kohavi, Tang and Xu 2020, Ch. 23; Hohnhold, O'Brien and Tang 2015). Signal to look for: engagement curves that peak in the first few days and decay thereafter.
**Primacy effect.** The mirror image. Users initially resist change because of friction, learning curves, and habit, so the short-run test underestimates the true long-run benefit. Common in complex UI redesigns and workflow changes where the user has invested in the old pattern. Signal to look for: flat or negative effects in week 1 that turn positive by weeks 3-4.
**Equilibrium effects.** The treatment changes the behaviour of the entire system, not just the treated units. In a two-sided marketplace, a feature that improves driver supply changes rider wait times, which changes rider retention, which changes future supply, and none of that is captured in a two-week user-level test. This is the mechanism that makes marketplace experimentation especially treacherous (Chamandy 2016; Kohavi, Tang and Xu 2020, Ch. 22). Signal to look for: experimental effects that do not replicate in geography-based or time-based switchbacks, or that show opposite signs across segments.
**Portfolio-level Winner's Curse (cumulative overestimation).** Even if every individual experiment were perfectly unbiased, the sum of individual lifts overstates the aggregate business impact. Features interact, cannibalise shared user attention, and are measured against denominators that other features have already moved. The aggregate is typically less than the sum of parts, and the gap is what the universal holdout is designed to measure. Signal to look for: aggregate business metrics that are flat despite a track record of positive experimental results.
:::
## Three ways your lift disappears
The notification feature's decay from +5% DAU to zero was not noise, and it was not bad luck. It was one of three distinct mechanisms that pull short-run treatment effects away from their long-run steady state. Each has a different causal structure, each leaves a different fingerprint in the data, and each has a different cost when it is misdiagnosed. This section is the taxonomy.
### Novelty effects
Users have a finite attention budget, and anything new commands a temporary, disproportionate share of it independent of the feature's true long-run value. That is the novelty effect. Users notice the new element, click it, explore it, and allocate weight to it in their mental model that is calibrated to its unfamiliarity, not its utility. The behaviour is not irrational, and it is not a measurement artefact. It is the efficient allocation of attention under uncertainty, and it decays as the feature becomes part of the background.
The decay rate is a function of exposure frequency, not calendar time. A daily-use feature (a notification badge, a home-screen widget, a search-bar redesign) exhausts its novelty budget within one to two weeks, because each user is exposed many times per day and habituates quickly. A feature encountered once per session, or once per week, may still be inside its novelty window months after launch, because each interaction still feels comparatively fresh. The standard two-week experiment is long enough to observe steady state on the first class of features and barely long enough to detect that the second class is still adjusting.
The diagnostic signal is unambiguous. Plot the daily treatment effect $\hat{\tau}(t)$ over the experiment window. If the estimate slopes downward, from a high value in the first days toward a lower asymptote, you are looking at novelty. The short-run average $\bar{\tau}$ overstates the long-run steady state $\tau_{\text{ss}}$ by an amount that depends on how front-loaded the curve is. That is exactly what happened to the notification feature: the day-1 lift was substantially larger than the averaged +5%, the daily lift by the end of the second week was already close to zero, and averaging over the window produced the headline number that later refused to reproduce in the aggregate metric.
Kohavi, Tang and Xu (2020, Ch. 23) group this under user-learned effects, the general class of behaviours that shift because users explore a new change more when it is first introduced. Hohnhold, O'Brien and Tang (2015) illustrate the business consequence at scale. In their Google Search ads experiments, increasing ad load produced a positive short-run revenue lift that dissipated over several months as users developed ads blindness, updating their propensity to click on ads downward in response to the lower average quality that a higher ad load imposes. The short-run experiment measured the immediate response. The long-run steady state, once user click propensity had reached its new equilibrium, was substantially lower, and in some launches the long-run revenue change was neutral or negative despite the strong short-run signal.
### Primacy effects
The primacy effect is the mirror image. Users have existing habits, mental models, and muscle memory built on the old design. A change disrupts those routines, and the short-run test captures the cost of unlearning them, not the value of the destination.
Consider a checkout redesign on a food delivery app that consolidates five screens into three, moves the payment selector, and relabels the confirmation button. Objectively, the new flow reduces clicks, form fields, and cognitive load, and first-time users convert on it at a higher rate than on the legacy design. But returning users have completed the old flow from muscle memory hundreds of times. In week one they hesitate at the new step, mis-tap the button, and abandon the cart. Conversion drops 2%. By week three, once returning users have internalised the new layout, conversion is running 1.5% above the old baseline and support tickets on payment errors are down. A two-week test window records the switching cost and misses the gain entirely.
Formally, if the day-$t$ treatment effect satisfies $\hat{\tau}(t) \to \tau_{\text{ss}}$ from below, the short-run average $\bar{\tau}$ understates the steady state. The diagnostic signal is a flat or negative daily treatment effect early in the window that rises over time, the exact inverse of the novelty pattern. Kohavi, Tang and Xu (2020, Ch. 23) document this as the second face of user-learned effects, where users are primed in the old feature and incur a relearning cost before reaching the new equilibrium.
Primacy is the more dangerous of the two mechanisms in practice, and the asymmetry is worth stating plainly. Novelty leads a team to overestimate a win. That mistake is embarrassing but visible: the launch fails to reproduce the promised numbers, the gap surfaces in the next quarterly business review, and someone is held accountable. Primacy leads a team to kill a feature that would have been a win. There is no quarterly business review moment for that failure. The feature is archived, the counterfactual revenue never appears on any dashboard, and the organisation continues to pay the friction cost the redesign was meant to remove. The loss is silent, and the organisational bias toward false negatives on complex UI changes is systematically larger than most data teams recognise.
One caveat on interpretation. A rising treatment effect curve is consistent with primacy, but it is also consistent with a feature that is simply bad and happens to look less bad over time. The primacy diagnosis is a hypothesis about the mechanism (returning users are relearning a workflow they had automated), and it needs to be defensible on that mechanistic ground before it is used to justify keeping a losing test alive. The curve is necessary evidence, not sufficient.
### Equilibrium effects
The third mechanism operates at a different level entirely. Novelty and primacy are about the same user's response changing over time. Equilibrium effects are about other users' outcomes changing because of the treated group's behaviour. The distinction is structural, not stylistic: novelty and primacy are longitudinal, within-user phenomena; equilibrium effects are cross-user phenomena that a within-user experiment cannot capture by construction.
Take a ride-hailing marketplace testing a heat-map improvement that helps drivers reposition toward emerging demand. Treated drivers reposition more efficiently, accept more trips, and earn more per hour. The user-level treatment effect on driver earnings is positive, stable, and well-measured. Now consider the second-order consequences. The treated drivers are no longer where the control drivers used to overlap with them, so control drivers pick up marginal trips they would not have won. Riders in some neighbourhoods see shorter ETAs, riders in others see longer ones. Surge pricing adjusts. Rider retention shifts on a timescale of weeks. The new supply-demand balance takes months to settle, and it does not respect the boundaries of the experiment.
This is interference in the sense discussed in [*When A/B Tests Lie*](https://joaovieira22.github.io/posts/AB%20testing%20com%20Spillovers/when_ab_tests_lie.html). The standard A/B design assigns drivers to treatment and control who are competing for the same rider pool, exactly the shared-resource case that Kohavi, Tang and Xu (2020, Ch. 22) catalogue alongside Uber/Lyft surge pricing, Airbnb inventory competition, and ad-campaign budget sharing. Chamandy (2016) develops the point for ridesharing specifically, where the coupling between treatment and control units through the marketplace makes the user-level treatment effect a poor guide to the global effect that would obtain once the whole system were on the new algorithm. The partial-equilibrium estimate and the general-equilibrium effect can differ in magnitude, and in genuinely two-sided markets they can differ in sign.
The diagnostic signal is not in the shape of the treatment effect curve. It is in the mismatch between a stable, well-measured user-level effect and the aggregate business metric. The driver experiment reports +5% per-driver earnings, and platform-wide gross bookings are flat. The curve is stable, so novelty and primacy are ruled out, and yet the launch does not reconcile with the top line. That discrepancy is the trace of an equilibrium effect. It cannot be diagnosed from a within-user time series, because the within-user time series is not wrong; it is answering the wrong question. Detection requires either a system-level comparison against the aggregate metric, cluster randomisation at the market or region level, or a switchback design that alternates the whole market between variants over time. A user-level randomised experiment, run for arbitrarily long, cannot see it.
This is the practical reason marketplace companies cannot rely on the same diagnostic toolkit as single-sided consumer apps. On a search engine or a social feed, a well-shaped daily treatment effect curve rules out most of what could go wrong. On a ride-hailing or food-delivery platform, a well-shaped curve rules out nothing about the effect that will actually show up on the P&L.
### Three signatures, one blind spot
Two of these mechanisms are diagnosable from data the experiment already produces. A declining daily treatment effect curve is novelty, and the short-run average is too high. A rising curve is primacy, and the short-run average is too low. In both cases, plotting $\hat{\tau}(t)$ against $t$ and looking at the slope is enough to raise the right suspicion, and the mechanism can be corroborated by asking whether the plausible causal story is habituation to a new stimulus or relearning of an old workflow.
Equilibrium effects are not in that curve. They are visible only when the user-level estimate is put next to a system-level metric, or when the experimental design itself is changed to randomise at the level of the interference. The asymmetry is worth internalising: novelty and primacy are diagnostic problems, and equilibrium effects are design problems. You cannot inspect your way out of the third one; you have to instrument for it before the experiment starts.
```{r}
#| label: fig-three-signatures
#| fig-cap: "Figure 1. Three shapes the daily treatment effect can take over a two-week window. Novelty starts high and decays toward the steady state. Primacy starts negative and rises toward the steady state. A stable true effect is shown for reference. Equilibrium effects do not appear on this chart, because they are not diagnosable from a within-user time series; a stable curve is consistent with an equilibrium effect of any magnitude."
#| fig-width: 8
#| fig-height: 4.5
library(ggplot2)
df <- data.frame(
day = rep(1:14, 3),
effect = c(
c(5.2, 4.6, 4.1, 3.7, 3.3, 3.0, 2.8, 2.6, 2.5, 2.4, 2.3, 2.3, 2.2, 2.2),
rep(2.0, 14),
c(-1.5, -0.8, -0.2, 0.3, 0.7, 1.0, 1.2, 1.4, 1.5, 1.6, 1.7, 1.7, 1.8, 1.8)
),
pattern = factor(rep(c("Novelty", "Steady-state", "Primacy"), each = 14),
levels = c("Novelty", "Steady-state", "Primacy"))
)
ggplot(df, aes(x = day, y = effect, color = pattern, linetype = pattern)) +
geom_line(linewidth = 1.2) +
geom_hline(yintercept = 0, linetype = "dashed", color = "grey50", linewidth = 0.5) +
scale_color_manual(values = c("Novelty" = "#e74c3c",
"Steady-state" = "#2ecc71",
"Primacy" = "#3498db")) +
scale_linetype_manual(values = c("Novelty" = "solid",
"Steady-state" = "dashed",
"Primacy" = "solid")) +
scale_x_continuous(breaks = 1:14) +
scale_y_continuous(limits = c(-2, 6), breaks = seq(-2, 6, 1)) +
labs(
title = "Daily treatment effect curves over the experiment window",
x = "Day of experiment",
y = "Estimated treatment effect (%)",
color = "Pattern",
linetype = "Pattern"
) +
theme_minimal(base_size = 13) +
theme(
plot.title = element_text(size = 14, face = "bold", margin = margin(b = 15)),
legend.position = "bottom",
panel.grid.minor = element_blank()
)
```
Even a team that diagnoses each of these three mechanisms correctly on every individual test still faces a fourth problem. The sum of many individually well-measured effects systematically overestimates the true cumulative business impact, not through decay of any single lift but through the way small, noisy estimates aggregate across a portfolio of launches. That is Section III, and the universal holdout is the design that finally lets us measure around it.
::: {.callout-note title="Three mechanisms, three signatures"}
- **Novelty effect.** Declining daily treatment effect curve. The short-run average overstates the long-run steady state.
- **Primacy effect.** Rising daily treatment effect curve, often starting negative. The short-run average understates the long-run steady state. This is the silent killer: the feature is tested and killed before the win materialises, and no quarterly review ever names the loss.
- **Equilibrium effects.** Cannot be seen in the daily curve at all. Diagnosis requires comparing the user-level estimate with the aggregate business metric, or a cluster-randomised or switchback design.
:::
## Why the sum of your wins is probably wrong
The first three mechanisms are per-test problems. Diagnose each one correctly, on every experiment, and you have solved the measurement problem for a single launch. You have not solved the measurement problem for a roadmap. That fourth problem, the portfolio-level Winner's Curse, is not a defect in any individual measurement. It is a property of the decision rule that sits on top of them, and it is the reason a team can be right about every experiment it ships and still be wrong about what it delivered this quarter.
Start from the single-experiment version. A launch decision conditioned on crossing a significance threshold systematically overestimates the true effect of the shipped variant, because you disproportionately ship the tests that got a lucky draw on top of whatever true effect they had. That is the Winner's Curse at the level of a single experiment, a statement about the expected value of the estimator conditional on shipping.
The portfolio-level claim is different, and the distinction is worth being pedantic about. It is not the claim that any individual test's reported lift is biased upward, though that is also true. It is the claim that when a team ships $N$ features in a quarter, each cleared through the same threshold-based decision rule, the sum of the $N$ measured lifts overestimates the sum of the $N$ true effects, and the gap does not shrink as $N$ grows. It compounds. Formally, letting $A$ denote the set of shipped experiments and $X_i$, $a_i$ the measured and true effects respectively,
$$\mathbb{E}\!\left[\sum_{i \in A} X_i\right] \;>\; \mathbb{E}\!\left[\sum_{i \in A} a_i\right],$$
with the gap accumulating monotonically in $|A|$. Shen and Lee (2017) derive this result explicitly for the Airbnb experimentation platform and show that every term in the difference is non-negative, because the expected value of a lower-truncated mean-zero distribution is strictly positive.
The intuition for why the gap compounds rather than cancels is the part that matters for anyone running a roadmap. Among the tests that clear the threshold, some are true positives with real but modest effects and some are noise draws that happened to land above the cutoff. The decision rule cannot tell them apart at the moment of shipping, because both look like a significant positive lift on the dashboard. Summing across many such decisions does not average the noise away, because the bias is one-directional by construction. Noise draws that fell below the threshold were never shipped, so they never enter the sum. What survives to the topline is a systematically upward-selected subsample of the underlying draws, and the more decisions you make under this rule, the more selection compounds. The problem is not any one test, it is what happens when you ship twenty of them.
The most concrete illustration of this in the public record is the Airbnb case documented by Shen and Lee (2017). Their team ran six experiments sequentially over several months, each of which crossed a significance threshold and was launched to all users, with a small split-holdout kept back to measure the aggregate impact directly. When the six individual point estimates were added up, the bottom-up sum on the target metric was 7.2 percentage points, while the split-holdout comparison measured the true aggregate impact at roughly 4 percentage points. Shen and Lee then applied their formal bias-adjustment for the selection effect directly to the bottom-up estimate, which brought it down to 5.3 percentage points, still well short of closing the gap to the holdout's 4.
Sit with the shape of that gap. On the same six launches, measured on the same metric, by the same team, using the same platform, the naive sum overstates the delivered impact by roughly a factor of two. Nothing about the six individual experiments was wrong. What was wrong was the assumption that you can add them.
```{r}
#| label: fig-airbnb-gap
#| fig-cap: "Figure 2. Three measurements of the same six Airbnb experiments. The naive bottom-up sum (7.2 percentage points) overstates the split-holdout's direct measurement (roughly 4.0) by about 2x. Applying the formal Winner's Curse bias-adjustment to the bottom-up sum brings it down to 5.3, still about 1.3x the holdout figure (Shen and Lee, 2017; GrowthBook, 2025)."
#| fig-width: 8
#| fig-height: 4.5
library(ggplot2)
df <- data.frame(
method = c("Bottom-up sum", "Bias-adjusted sum", "Split-holdout"),
impact = c(7.2, 5.3, 4.0)
)
df$method <- factor(df$method,
levels = c("Bottom-up sum", "Bias-adjusted sum", "Split-holdout"))
ggplot(df, aes(x = method, y = impact, fill = method)) +
geom_col(width = 0.6, color = "black", linewidth = 0.3) +
scale_fill_manual(values = c("Bottom-up sum" = "#e74c3c",
"Bias-adjusted sum" = "#f39c12",
"Split-holdout" = "#2ecc71")) +
geom_text(aes(label = sprintf("%.1f pp", impact)),
vjust = -0.5, fontface = "bold", size = 4.5) +
scale_y_continuous(limits = c(0, 8.5), breaks = seq(0, 8, 1)) +
labs(
title = "Airbnb aggregate impact: three ways of measuring the same six launches",
x = NULL,
y = "Aggregate impact on target metric (pp)"
) +
theme_minimal(base_size = 13) +
theme(
plot.title = element_text(size = 14, face = "bold", margin = margin(b = 15)),
axis.text.x = element_text(size = 11, angle = 0, hjust = 0.5),
axis.title.y = element_text(size = 12, margin = margin(r = 10)),
panel.grid.major.x = element_blank(),
panel.grid.minor = element_blank(),
legend.position = "none"
)
```
Etsy formalised the same insight operationally. Shao and Burke (2022) describe how the team stopped trusting the bottom-up sum of individual experiment results and instead runs a quarterly "comparison period" in which a holdout cohort continues to see the pre-quarter baseline while the treatment cohort is exposed to the combination of all adopted treatments at once. The total incremental impact is measured directly against that holdout rather than reconstructed by summation. Shao and Burke are careful about the framing. The point is not to attach a headline overestimation figure to any particular quarter, it is to make the structural observation that the combined effect of many treatments, measured directly, is generally not equal to the sum of their individually measured effects. That gap, in either direction, is exactly the object the portfolio-level Winner's Curse is asking about.
This is why the section's central point matters more than any single figure. Correcting for novelty, primacy, and equilibrium effects on every individual test, even perfectly, does not touch the portfolio-level Winner's Curse. The bias is not in the measurement of any one launch. It lives in the ensemble of shipping decisions, in the fact that the very rule you use to select launches induces upward selection on the estimates you then sum. A team can diagnose each of the three per-test mechanisms correctly, on every test, and still end the quarter with a bottom-up roadmap number that does not survive contact with the holdout. Every individual launch was measured correctly is not the same claim as the roadmap moved the topline number, and that gap is the conversation a product leader is going to have in every QBR until the measurement architecture changes.
Two families of correction address this, and they act at different levels. Shrinkage estimators attack the problem at the single-test level by pulling extreme estimates toward a prior, damping the Winner's Curse before the launch decision is made. The holdout-based correction described by Martini (2023), and instantiated in the Airbnb and Etsy cases, is the portfolio-level answer. Rather than summing individual lifts, you measure the joint effect of all shipped variants against a group that never saw any of them, and you take that measurement as the truth.
We now have all four mechanisms on the table: novelty, primacy, equilibrium effects, and the portfolio-level Winner's Curse. Each of them, on its own, is a reason a lift measured today may not survive contact with next quarter's business review. Section IV turns to the proposed answer, the universal holdout, and explains why it addresses all four at once rather than requiring a separate fix for each.
::: {.callout-important title="The portfolio-level Winner's Curse, in one line"}
Fixing novelty, primacy, and equilibrium effects on every individual test does not fix the portfolio-level Winner's Curse. The bias lives in the ensemble of shipping decisions, not in any single test's measurement, and threshold-based selection guarantees it compounds with the number of launches rather than averaging out. Every individual test can be measured correctly and the roadmap total can still be wrong.
:::
## What is a universal holdout?
Novelty, primacy, equilibrium effects, and the portfolio-level Winner's Curse all describe different ways a lift measured in a two-week window fails to survive to a quarterly business review, and each of the fixes discussed so far attacks one mechanism at a time. The universal holdout is a different move: rather than diagnosing which mechanism is at play in any given test, it changes what is being measured, and in doing so substantially reduces exposure to all four at once.
A universal holdout is a group of users held permanently in control for a defined period, usually a full quarter or half-year, receiving none of the features shipped during that period. The contrast with a standard A/B test control group is not one of scale, it is structural. A standard control group is scoped to a single feature and is released once that feature's test concludes. A universal holdout persists across every feature shipped during the period, across every team, across every individual experiment on the platform. It is not another feature-level control group, it is a control group of control groups.
What the holdout measures is a single quantity, taken at the end of the period: the difference between users in the holdout and users on the treated platform. That difference is a direct estimate of the true cumulative effect of everything shipped, net of novelty decay, net of primacy adjustment, net of the portfolio-level Winner's Curse, and (with the important caveats below) largely net of within-user equilibrium reallocation. The reason all four are largely defused is that they never enter the measurement in the first place, because it is a single long-run comparison rather than a sum of many short-run point estimates.
### Why it addresses all four mechanisms
Take the mechanisms one by one.
Novelty and primacy both distort the endpoint of a short test. The holdout comparison happens at the end of a six-month period, not at the two-week mark, so both effects have had time to resolve into their steady-state values. The holdout does not need to determine which one was operating on any given feature, it simply measures the outcome after enough time has passed for both to settle. This is the same logic behind post-period and time-staggered designs in the long-term experimentation literature (Hohnhold, O'Brien and Tang, 2015; Kohavi, Tang and Xu, 2020, Ch. 23), applied at the portfolio level rather than to one feature at a time.
Equilibrium effects are more nuanced. A short-run user-randomised test cannot see system-level reallocation: if treatment users book more listings, take more rides, or consume more ad inventory, the control side of the same test loses inventory to them, and the measured delta is contaminated by the reallocation rather than by the feature's own value (Kohavi, Tang and Xu, 2020, Ch. 22). A universal holdout is a real, persistent population living inside the same system as the treated population, so many supply reallocation, wait-time shifts, or ecosystem responses that affect the treatment side are visible in the comparison at the endpoint. That said, a user-level holdout does not eliminate marketplace interference, it inherits it: as Section VI discusses in the context of ratcheting, the holdout population itself can be systematically disadvantaged by the treated population's behaviour. The holdout does not have to sit outside the marketplace to be uncontaminated by it, it has to sit outside the roadmap, and marketplace platforms typically require a cluster-level or geo-level design instead.
The portfolio-level Winner's Curse is where the argument turns sharpest. As covered in Section III, summing $N$ individually significant point estimates inherits $N$ draws of the same one-directional selection bias, which is why Airbnb's bottom-up sum of 7.2 percentage points was still well above the true, split-holdout-measured impact of roughly 4 points even after the formal bias-adjustment brought it down to 5.3 (Shen and Lee, 2017). A universal holdout is one measurement of the whole period's effect, not a sum of $N$ individually significant results, so the selection bias never enters. It measures the sum directly rather than summing the measurements.
This is the section's central claim: the universal holdout does not correct any individual mechanism, it replaces the need to diagnose or correct for them with a single direct comparison at the endpoint. Correction is per-test work. The holdout replaces per-test work with one direct measurement. That is a stronger property than having four separate fixes, because a team running per-test corrections has to get every diagnosis right on every launch. A team running a universal holdout only has to keep the control group stable and wait.
None of this makes the universal holdout a universal solution. In marketplaces, ads systems, ranking pipelines, and networked products, the design assumptions under which the holdout is an unbiased estimate of cumulative impact are precisely the assumptions those systems tend to break: interference, continuous model retraining, and shared-resource dynamics can all leave the holdout population systematically disadvantaged in ways that show up as a bias in the measured delta, not just as noise. Section VI is where those failure modes are named specifically. The claim in this section is narrower and worth holding onto: where the design assumptions hold, one direct comparison at the endpoint is a stronger instrument than four separate per-test corrections stacked on top of each other.
None of this makes the universal holdout a silver bullet, and it is worth being explicit about that before Section VI's cost section forces the point anyway. It solves the measurement problem at the portfolio level; it does not solve the design problem at the interference level, which is why marketplace and social products often cannot use one in its user-randomised form at all. And it does not answer the question of whether any specific feature is genuinely working, which is Section V's job. What the universal holdout does is give a product organisation a defensible answer to the one question that keeps recurring at QBRs and half-cycle reviews: what is the true cumulative effect of everything we shipped this period.
### How Meta runs it
The clearest operational example comes from Meta. Madhusudanan (2022) describes how most product teams on the core Facebook app calculate the cumulative impact of every feature shipped over the last six months, and align that measurement with their goal-setting and performance review cycle. At the start of each half, they create a small holdout, typically 1-5% of users, hold it out from every feature shipped during that half, and at the end compare it against users who received everything the teams launched. Then they release the holdout and start a new one for the next half.
The organisational point matters as much as the statistical one. A holdout that feeds into performance reviews and half-cycle goal-setting has to be defensible to people outside the experimentation team, which is a different bar than a holdout used for internal measurement only. A product leader has to be able to explain, to a finance partner or a general manager, why 1% of users saw none of last half's launches. That defensibility comes from the power calculation, not from intuition, which is what makes the sizing question the next thing worth pinning down.
### Sizing the holdout
The size of the holdout, and why 1-5% is a defensible range rather than a suspiciously small one, comes from a specific power calculation. Martini (2023) sets out the key principle: the holdout is powered over the expected cumulative effect of a full period's worth of shipped features, not over any single feature's expected effect size. This is a categorically different calculation from the power analysis used for single-experiment control groups against single-feature lifts.
The intuition for why the numbers come out so much smaller is straightforward. The cumulative effect of dozens of shipped features compounds. Even if each one moves a metric by a few tenths of a percent, the aggregate effect over a half is meaningfully larger, and meaningfully more detectable, than any single feature's contribution. A holdout percentage that would be badly underpowered for one feature is comfortably powered for the sum. That is precisely why 1-5% works at Meta's scale, despite being nowhere near what a single-feature test would tolerate.
Scale is a real constraint on the lower end of that range. Martini (2023) is direct about this: without something close to Facebook's traffic volume, a 1% holdout is unlikely to be powered for the cumulative effect it needs to detect, and an organisation in that position is better off not running one at all. For most organisations, the operational question is not whether 1% is defensible in principle, it is whether the aggregate half-year effect a team expects to ship is large enough for whatever percentage they can afford to withhold. The right number is whatever the power analysis produces given the expected cumulative lift and the platform's traffic, not a fixed constant borrowed from someone else's blog post.
The universal holdout answers the portfolio-level question with real precision: what is the true cumulative effect of everything the organisation shipped this period. But it does not answer a narrower question a team often needs to answer about one specific feature, whether that one launch is genuinely working, whether its short-run read can be trusted, or whether its true value is only visible on a longer timescale. That distinction, between the universal holdout and its feature-level counterpart, is Section V.
::: {.callout-tip title="Universal holdout, in one line"}
A universal holdout is a small group of users, typically 1-5% at platform scale, held out from every feature shipped in a defined period (usually a quarter or half-year) and compared against the treated population at the end of that period. It measures the true cumulative effect of the roadmap in a single direct comparison rather than by summing individual test results.
:::
## Feature-level holdouts: when individual experiments need more time
Section IV set aside a narrower question than the one the universal holdout answers: what if you do not want to measure everything shipped this quarter, but one specific feature whose short-run experiment result you have a concrete reason to distrust? That question calls for a different instrument. A universal holdout measures the cumulative effect of everything shipped over a period, across every team and every feature. A feature-level holdout measures one specific feature, held out for longer than a standard experiment window, when there is reason to suspect the short-run result is wrong in a particular way: either a novelty effect that has not yet decayed, or a genuine effect that takes time to build. The second case is the mirror of primacy, the benefit is real, but it compounds slowly rather than appearing immediately.
Structurally, a feature-level holdout is the same control-group design as a standard A/B test. The mechanics are identical. A fraction of users is randomly assigned to a control group that does not receive the feature, everyone else does, and outcomes are compared. There is no new statistical machinery here. The only differences are duration and intent. A two-week test estimates the short-run treatment effect under the tacit assumption that the short-run number is decision-relevant. A feature-level holdout keeps the control group open for weeks or months longer because the team has a specific, articulable hypothesis about why that assumption fails for this feature. The question is not whether the feature works — it is whether the standard window was long enough to tell.
### The Facebook game streaming case
The best documented example of this pattern comes from Facebook's game streaming rollout. Madhusudanan (2022) describes a test that invited people to join a streamer's community while watching that streamer's video. Four weeks into the test, the topline business metrics were neutral. More people were joining communities, but the metrics the team was actually trying to move had not budged.
The team faced a familiar decision. Trust the four-week read and shelve the feature, or trust their conviction that community-building was fundamentally the right experience for users and ship it anyway. They chose the second path, launching the feature to almost everyone while keeping a small holdout in place. Four months later, that holdout measured a double-digit increase in topline metrics from the feature (Madhusudanan, 2022).
The mechanism is transparent once you see it. Community-building behaviour compounds slowly. A user who joins a streamer's community in week one does not produce measurable business value until that community reaches a critical mass of activity, and reaching critical mass takes time. The four-week window captured the cost and neutrality of the early adoption phase, not the value that accumulated as communities matured. This is a slower-building cousin of the primacy effect from Section II, operating on a timescale of months rather than days or weeks. Kohavi, Tang and Xu (2020, Ch. 23) frame the same idea in more general terms. They give four standard reasons short-run and long-run treatment effects diverge (user-learned effects, network effects, delayed experience and measurement, and ecosystem change) and treat 3+ months as their working definition of long term.
The practical takeaway is not that four-week tests are useless. It is that shipping on a neutral four-week read was the right call in this specific case, precisely because the team kept a holdout in place to confirm it later. Without the holdout, there would have been no way to distinguish a genuinely neutral feature from one whose value simply had not manifested yet. The holdout is what turned a leap of faith into a measured bet with a verification mechanism attached.
### When to use each type of holdout
The decision rule is sharper than "long-term effects need holdouts."
Use a universal holdout when the goal is measuring the cumulative impact of a full roadmap or period, for goal-setting, performance reviews, or answering "did our shipped work move the business." The unit of analysis is the portfolio, not any single feature.
Use a feature-level holdout when one specific feature has a concrete, articulable reason to distrust its short-run read. That reason typically falls into one of two buckets. The first is novelty inflation, where users are engaging because the feature is new and that engagement will decay once it becomes normal. The second is the mirror image, a slow-building effect like the game streaming case, where the true value only manifests after a user-learning, behavioural, or network effect has had time to develop. Kohavi, Tang and Xu (2020, Ch. 23) list the same triggers (user-learned effects, network effects, delayed experience and measurement, and ecosystem change) as the situations where a short-run estimate is answering the wrong time-indexed question.
Feature-level holdouts are the exception, not the default. Most features do not need one. If a feature triggers frequently, delivers immediate value, has no plausible user-learning path, and has no network or marketplace component, the standard test window should be enough. Running a feature-level holdout for every launch would impose unsustainable engineering overhead and opportunity cost. Reserve them for cases where the team can write down, in one sentence, the specific mechanism causing the short-run bias, before going looking for a more convenient answer four months later. An articulable reason is the price of entry. If you cannot state why two weeks is too short, you do not have a justification for a holdout, you have a lack of conviction in the feature itself.
### The sizing trade-off
Sizing a feature-level holdout is a business decision as much as a statistical one. The choice is between a smaller holdout (around 1% of users) held for a longer period and a larger holdout (around 10% of users) held for a shorter period.
The smaller, longer holdout costs less in immediate opportunity cost, because fewer users are denied the feature at any given time, but it delays the answer. The larger, shorter holdout answers faster, because the comparison accumulates information more quickly, but it withholds the feature from a larger slice of the user base while the question remains open.
Two business inputs govern the choice. How costly is it to delay the shipping decision? If the feature is strategically urgent, or if downstream roadmap work depends on knowing the answer, a larger holdout that closes out sooner is worth the temporary opportunity cost. How costly is it to withhold the feature in the meantime? If the feature clearly improves the user experience, or if it affects revenue, retention, or safety metrics that compound daily, a smaller holdout that leaves more users on the treatment is the cheaper design. Madhusudanan (2022) notes that one of Facebook's most expensive holdouts is a long-term ads holdout, in which a set of users continue to use the product without advertisements. It is expensive precisely because the withheld ad revenue is large, and it is tolerated precisely because the measurement it produces, the cost of ads on engagement, is more valuable to the business than any cheaper substitute.
Power analysis alone cannot resolve this. The statistical question, whether you can detect the effect you care about at a given holdout size and duration, is necessary but not sufficient. Kohavi, Tang and Xu (2020, Ch. 23) describe holdback designs of roughly the same shape as the two options above, with 10% control kept live for weeks or months after a 90% launch as the canonical case, and they are explicit that the design tension is opportunity cost against measurement sensitivity, not one against the other in isolation. There is no universal percentage. The right design follows the business cost structure of the specific feature, not a generic experimentation template.
Both universal and feature-level holdouts sound straightforward in principle, but running either one in practice carries real costs that a team has to accept before committing to the approach. Section VI covers those costs directly: engineering overhead, opportunity cost, monitoring risk, and user experience friction.
::: {.callout-tip title="Universal or feature-level, in one line"}
Use a universal holdout to measure the cumulative impact of everything shipped over a period. Use a feature-level holdout when one specific feature has a concrete, articulable reason to distrust its short-run read, either because the effect might decay (novelty) or because it might take months to build (the slow-building cousin of primacy). Most features do not need one.
:::
## The costs you need to accept before running a holdout
Everything up to this point has argued that holdouts are the correct instrument for cumulative long-run impact, and that argument stands. But it is not the whole picture. Any reader tempted to walk into a Monday planning meeting and propose "let's just run a universal holdout" should first read what follows. Madhusudanan (2022) is blunt on this point: "holdouts are not cheap." The four costs below (engineering overhead, opportunity cost, integrity monitoring, and user experience asymmetry) are why most companies either never launch a holdout or abandon the one they have after a cycle or two. A holdout launched without a plan for these costs is not a neutral experiment. It produces data that looks valid, gets summarised into a slide, and quietly misleads the organisation for a quarter. Martini (2023) is explicit that this outcome is worse than not running a holdout at all. A reader who decides against a holdout after reading this section has made a legitimate, well-informed decision.
### Cost 1: engineering overhead
Every feature shipped while a holdout is active must maintain a live if-else fork in the codebase for the entire holdout duration. A team shipping fifty features over a quarter is maintaining fifty active branches simultaneously, each of which must correctly identify holdout users and route them to the pre-treatment path. The test and debug matrix grows with velocity, not with team size, so fast teams pay this cost fastest. Madhusudanan (2022) reports the concrete case: when Instagram needed to resize its ad holdout, the change took months to validate and launch, with hacky changes across multiple codebases. The business consequence was not only engineering time but the risk of compromising the main holdout's integrity during the transition.
The threshold at which this cost becomes infeasible is organisational, not statistical. If your team cannot commit to a focused engineering push at holdout retirement, when the branches are collapsed and the code is cleaned up in a single coordinated pass, the technical debt will compound faster than the measurement is worth. Distributed cleanup, where each engineer is asked to clean up their own feature flag individually, produces orphaned code paths that linger for quarters and quietly break future experiments.
### Cost 2: opportunity cost
Users in the holdout group receive a worse product for the entire measurement period. That is the point, and it is also the price. Facebook's long-term ads holdout, cited by Madhusudanan (2022), means a set of users genuinely uses the product without advertisements; the company treats this as valuable information, but the revenue foregone is real. The magnitude has a simple closed form. Let $h$ be the holdout fraction of users, $\delta$ the estimated true cumulative lift, and $R$ the daily revenue on the treated share. Then the daily opportunity cost is
$$\text{Daily cost} = h \times \delta \times R.$$
For $h = 0.02$, $\delta = 0.03$, and $R = \$1{,}000{,}000$ per day, the arithmetic is $\$600$ per day, or roughly $\$54{,}000$ over a 90-day holdout. Substitute your own numbers; the formula is the same. The holdout fraction $h$ should be set using the sizing rule from Section IV, balancing detection power against this daily charge.
The cash figure is usually small relative to the measurement value, but a second cost does not appear in the formula. Holdout users who see friends with features they do not have generate customer support tickets and negative sentiment (Madhusudanan, 2022). That social cost scales with product visibility rather than with $h$, and it is what typically forces a holdout to be retired earlier than the sizing formula would suggest.
### Cost 3: integrity monitoring
A broken control variant, where holdout users accidentally receive a treated feature, is strictly worse than having no holdout, because the resulting estimate is biased downward and still looks statistically valid. Madhusudanan (2022) is explicit: contamination affecting only one percent of your users can make the entire holdout useless if it is caught only at the end of the holdout period. Monitoring is therefore not optional. It requires three components:
- Automated checks that every shipped feature correctly excludes holdout users, run on every release.
- A regular cadence review (Madhusudanan recommends monthly at minimum) that flags unexpected metric alignment between holdout and treatment as a contamination signal.
- A documented process for hotfixes and security patches, which must reach every user including the holdout, because holding back security fixes is not defensible.
Martini (2023) reinforces the operational point: the detection system must exist before launch, not after.
::: {.callout-warning title="A contaminated holdout is worse than no holdout"}
A holdout where control users accidentally receive treatment features produces a systematic underestimate of the true cumulative effect, and the bias is difficult to detect without dedicated monitoring (Martini, 2023). Unlike a missing experiment, where you simply have no data and know it, a contaminated holdout produces data that looks valid but is systematically wrong, and the organisation acts on it. Build the contamination detection system before you launch the holdout, not after.
:::
### Cost 4: user experience asymmetry in networked products
The three costs above are largely operational. The fourth is structural, and it is the reason marketplace and platform companies most often decide against a universal holdout. In a networked product, holdout users interact with treated users. On a ride-hailing platform, a holdout driver without repositioning features receives fewer trip requests because treated drivers are better placed for high-demand zones; the "control" driver's earnings fall not because a feature was directly withheld from them, but because the treated population is absorbing the supply-side surplus. The holdout is then measuring a mixture of the feature's own effect and the competitive disadvantage imposed on the control group.
Martini (2023) names the extreme case: **ratcheting**, where each individual experiment appears to increase metrics, but every gain in the treatment group comes at the expense of the control group. Both the individual experiments and the holdout that aggregates them are invalid in this regime. Formally, ratcheting is a violation of the Stable Unit Treatment Value Assumption. Under the Rubin causal model, unit $i$'s outcome should depend only on its own assignment, $z_i$, not on the full assignment vector $z$ (Kohavi, Tang and Xu, 2020, Eq. 22.1). Marketplace interference breaks this identity exactly. This is the same mechanism covered in [*When A/B Tests Lie*](https://joaovieira22.github.io/posts/AB%20testing%20com%20Spillovers/when_ab_tests_lie.html).
### When not to run a holdout
Four scenarios where the honest answer is: do not run one. A VP of Engineering should be able to classify their situation in under a minute.
- **Infrastructure changes and bug fixes.** Supporting old and new infrastructure in parallel is expensive, and knowingly withholding a bug fix from a slice of users ships them a broken product (Madhusudanan, 2022).
- **Cross-user features.** Collaborative editing, social graphs, and marketplace matching all require that a user's counterparts also have the feature (Madhusudanan, 2022). A user-level holdout breaks the feature for holdout users and for their counterparts on the other side of the interaction. The correct design is a cluster-level holdout at the organisation, city, or market level.
- **ML models with continuous retraining.** A frozen model held as the control is systematically handicapped by stale training data, and the comparison against a continuously retrained treatment is biased in favour of treatment regardless of the underlying feature quality (Martini, 2023). This is a bias problem, not a noise problem, and running the holdout longer makes it worse.
- **Non-persistent user identifiers.** For logged-out users, guest sessions, or any assignment that relies on cookies, it is nearly impossible to guarantee that a user is truly held back over months (Martini, 2023). Cookies expire, devices change, and the holdout leaks. Reserve holdouts for experiments with stable targeting.
### The alternative when holdouts are not feasible
Two practical substitutes exist. **Geo-based holdouts** hold an entire city or market in control, which eliminates cross-user contamination at the cost of a smaller effective sample and reduced statistical power. **Backtests** (Madhusudanan, 2022) remove a feature from a small set of users after full rollout and compare their metrics to everyone else; they work well for non-user-visible infrastructure changes and for validating that a positive result reproduces, but they are not a substitute for a prospective holdout when the goal is cumulative measurement. Where neither is feasible, the shrinkage correction from Section III remains the honest fallback: it will not recover the true long-run effect in the presence of interference or model staleness, but it will stop the naive bottom-up sum from being reported as if it were.
Section IV gave the sizing rule and Section V gave the feature-level design; this section has given the invoice. Section VII moves from decision to implementation, simulating a holdout in R, estimating the cumulative effect against the naive bottom-up sum, and showing what the correction looks like on data you can inspect line by line.
## Full R implementation: simulating and analysing holdout data
Having weighed those costs, the natural next question is whether the mechanisms described in Sections I to III actually reproduce, numerically, once we sit down and simulate them. They do, and the simulation itself is short enough to fit in a single R script. What follows is that script, broken into blocks with the reasoning in between, so that any reader with R installed can reproduce the two headline patterns of this post: a short-run read that overstates a decaying novelty effect's true steady state (Sections I to II), and a bottom-up sum of individual experiment effects that overstates a holdout-measured cumulative effect (Section III).
::: {.callout-note title="This simulation is illustrative, not empirical proof"}
The mechanisms below are encoded intentionally into the data-generating process to demonstrate the consequences discussed in Sections I to III. The gap between the early read and the late read, and the gap between the bottom-up sum and the holdout estimate, follow directly from the parameters chosen (novelty amplitude, decay rate, interaction dampening, selection-conditioned bias). The simulation shows what these mechanisms imply once they exist; it does not prove that they exist in any given real-world experiment. The empirical evidence for that lives in the Airbnb, Etsy, and Meta cases discussed in Sections III to V.
:::
The setup is deliberately minimal (base R plus dplyr and ggplot2), and every subsequent step reuses the same simulated data. Nothing is re-simulated inside a later block, which means the numeric relationships between blocks are real relationships in the same dataset, not artefacts of independent draws.
```{r}
#| label: setup
#| message: false
#| warning: false
library(dplyr)
library(ggplot2)
# Named seed: date of this post, for reproducibility
set.seed(20260705)
```
### Simulating a 90-day holdout with a decaying novelty effect
The functional form used here follows the novelty-decay pattern described in Kohavi, Tang and Xu (2020, Ch. 23), where a user-learned effect starts elevated and converges toward an asymptotic steady state. Concretely, the true treatment effect on day $t$ is modelled as
$$\tau(t) \;=\; \tau_{\text{ss}} \;+\; A \cdot \exp(-t / \lambda),$$
where $\tau_{\text{ss}}$ is the steady-state lift, $A$ is the novelty amplitude, and $\lambda$ controls the decay speed (the number of days after which the novelty component has fallen to $1/e \approx 37\%$ of its day-zero value). Around this true effect, daily observed lifts are perturbed by sampling noise, so that the simulated series resembles what an analyst would actually see rather than a noiseless curve. A reader who wants to sharpen or flatten the diagnostic curve from Section II only needs to change `A` or `lambda` and re-run.
```{r}
#| label: sim-novelty
n_days <- 90
steady_state <- 0.010 # 1.0% true long-run lift
novelty_amp <- 0.030 # 3.0 pp additive novelty on day 1
lambda <- 20 # decay constant (days)
daily_noise_sd <- 0.004 # daily sampling noise around the true effect
daily <- tibble(
day = 1:n_days,
true_effect = steady_state + novelty_amp * exp(-day / lambda),
noise = rnorm(n_days, mean = 0, sd = daily_noise_sd),
obs_effect = true_effect + noise
)
head(daily, 5)
tail(daily, 5)
```
The `true_effect` column is the curve the analyst is trying to recover, and `obs_effect` is what they actually see. Because the novelty component decays geometrically, the first few rows sit well above `steady_state` and the last few rows fluctuate around it.
### Comparing the day-1 read to the steady-state read
A standard short-run A/B test read corresponds to taking the mean of the first few days of `obs_effect`, before novelty has decayed. The steady-state read corresponds to the mean of the final days of the window, once the exponential term is close to zero (by day 76, $\exp(-76/20) \approx 0.022$, so the novelty component contributes less than 0.07 percentage points to the true effect). Comparing them numerically reproduces the diagnostic signature discussed in Section II, a monotonically declining daily treatment effect curve, without having to derive it again.
```{r}
#| label: early-late
early_read <- daily |>
filter(day <= 3) |>
summarise(mean_effect = mean(obs_effect)) |>
pull(mean_effect)
late_read <- daily |>
filter(day >= 76) |> # last 15 days
summarise(mean_effect = mean(obs_effect)) |>
pull(mean_effect)
ratio_early_late <- early_read / late_read
list(
early_read = round(early_read, 4),
late_read = round(late_read, 4),
ratio_early_late = round(ratio_early_late, 2)
)
```
The qualitative pattern the reader should expect, and the one this simulation is designed to expose, is `early_read > late_read`, with the exact ratio dictated by the amplitude and decay speed above. That gap is exactly the mechanism Section II attributed to Kohavi, Tang and Xu (2020, Ch. 23): the short-run read is not measuring the steady state, it is measuring the steady state plus a novelty term that has not yet decayed.
### Comparing the bottom-up sum to the holdout estimate
To reproduce the portfolio-level Winner's Curse from Section III, the simulation now represents a sequence of six experiments, each with its own true effect and its own sampling noise, mirroring the Airbnb case referenced there. The bottom-up sum aggregates the individually estimated effects the way a launch tracker or an "insights" dashboard usually does. The holdout estimate, by contrast, is a single direct comparison between a combined-treatment population that has been exposed to all six shipped features and a stable universal holdout population that has been exposed to none of them. This is the "stack everything into one test" logic that GrowthBook (2025) argue captures the cumulative and interactive effects that a bottom-up sum cannot.
Two forces are encoded explicitly in the simulation. First, each individual estimate carries a small selection-conditioned upward bias, because the experiments that ship are precisely the ones whose short-run numbers landed high in noise. Second, the combined true effect is not the sum of individual true effects; it is dampened by an interaction term to reflect the diminishing returns of shipping six overlapping features in the same product surface. These are the imposed properties that produce the headline gap, not empirical estimates, and the reader should read the resulting ratio as an illustration of the mechanism rather than as a prediction of any particular team's experience.
```{r}
#| label: sim-portfolio
n_experiments <- 6
n_per_arm <- 20000 # users per arm, per experiment
true_effects <- c(0.012, 0.008, 0.010, 0.006, 0.009, 0.007) # per-experiment true lifts
# Individually estimated effects, one experiment at a time
per_exp <- tibble(
experiment = seq_len(n_experiments),
true_effect = true_effects,
# Each individual estimate: mean(treatment) - mean(control), with noise
# plus a small positive selection-conditioned bias (winners are the ones we look at)
est_effect = true_effects +
rnorm(n_experiments, mean = 0, sd = 0.002) +
abs(rnorm(n_experiments, mean = 0, sd = 0.001))
)
bottom_up_sum <- sum(per_exp$est_effect)
# Holdout: one direct comparison, combined-treatment vs. universal holdout.
# The combined-treatment true effect is NOT the sum of individual effects,
# because of diminishing returns and interactions between features.
interaction_dampening <- 0.65 # combined effect is ~65% of the naive sum
combined_true_effect <- interaction_dampening * sum(true_effects)
treatment_outcomes <- rnorm(n_per_arm, mean = combined_true_effect, sd = 1)
holdout_outcomes <- rnorm(n_per_arm, mean = 0, sd = 1)
holdout_estimate <- mean(treatment_outcomes) - mean(holdout_outcomes)
ratio_naive_holdout <- bottom_up_sum / holdout_estimate
list(
bottom_up_sum = round(bottom_up_sum, 4),
holdout_estimate = round(holdout_estimate, 4),
ratio_naive_holdout = round(ratio_naive_holdout, 2)
)
```
The pattern to expect is `bottom_up_sum > holdout_estimate`, driven by the two forces above. GrowthBook (2025) report that in the Airbnb case they cite, the naive sum overstated a holdout-measured impact by roughly 2x, and a bias-corrected sum still overstated it by roughly 1.3x. Those multiples are the source's own reported figures for one documented case, useful here as a benchmark for interpreting whatever ratio this simulation happens to produce on a given run, not as a guaranteed outcome.
### Computing a confidence interval for the true cumulative effect
For the holdout comparison, a two-sample $t$ interval is the natural choice: the two arms are independent, the sample sizes are large, and the outcome is close enough to Gaussian that Welch's approximation is essentially exact. A block bootstrap would be preferable if the outcome were heavily skewed or if there were meaningful within-arm dependence across days, but neither is present in this simulated setting.
```{r}
#| label: ci
ci_test <- t.test(treatment_outcomes, holdout_outcomes, var.equal = FALSE)
list(
point_estimate = round(unname(ci_test$estimate[1] - ci_test$estimate[2]), 4),
ci_lower = round(ci_test$conf.int[1], 4),
ci_upper = round(ci_test$conf.int[2], 4),
ci_width = round(diff(ci_test$conf.int), 4)
)
```
The practical logic follows the same cumulative-monitoring principle Uber's Experimentation Platform team describes for their sequential tests: confidence intervals around a cumulative difference narrow as more days of data accumulate, which is why a well-run holdout is read once the interval is finally informative, not continuously peeked at from day one. Uber's own platform separately supports what its engineers call a universal holdout, used explicitly to measure the long-term effect of everything shipped in a given domain (Uber XP Team, 2018).
### Visualisation
The plot below is the section's single most shareable artifact, the one to screenshot into a stakeholder deck when explaining why a day-1 read is not a launch decision. It shows the observed daily effect, the true decaying curve underneath it, and the steady-state line the curve is converging toward. The shaded regions mark the early-read window (days 1 to 3) and the late-read window (days 76 to 90) used in the comparison above.
```{r}
#| label: fig-novelty-decay
#| fig-cap: "Figure 3. Novelty decay in a 90-day simulated holdout window. Observed daily lift (grey points), true decaying effect (blue curve), and steady state (dashed line). The novelty component decays visibly over the first three to four weeks and then flattens toward the steady-state line."
#| fig-width: 8
#| fig-height: 4.5
ggplot(daily, aes(x = day)) +
annotate("rect", xmin = 1, xmax = 3, ymin = -Inf, ymax = Inf,
alpha = 0.15, fill = "#d1495b") +
annotate("rect", xmin = 76, xmax = 90, ymin = -Inf, ymax = Inf,
alpha = 0.15, fill = "#2e8b57") +
geom_hline(yintercept = steady_state, linetype = "dashed", colour = "grey40") +
geom_point(aes(y = obs_effect), colour = "grey60", size = 1.3) +
geom_line(aes(y = true_effect), colour = "#1f4e79", linewidth = 1.1) +
annotate("text", x = 2, y = max(daily$obs_effect),
label = "early read", hjust = 0, vjust = 1,
colour = "#8b1a2b", size = 3.6) +
annotate("text", x = 83, y = max(daily$obs_effect),
label = "late read (steady state)", hjust = 0.5, vjust = 1,
colour = "#1c5c3a", size = 3.6) +
annotate("text", x = 60, y = steady_state,
label = "steady-state lift", vjust = -0.6,
colour = "grey30", size = 3.4) +
scale_y_continuous(labels = scales::percent_format(accuracy = 0.1)) +
labs(
x = "Day of holdout window",
y = "Estimated treatment effect",
title = "Novelty decay in a 90-day holdout",
subtitle = "Observed daily lift (grey), true decaying effect (blue), steady state (dashed)"
) +
theme_minimal(base_size = 12)
```
A reader who wants a sharper or gentler diagnostic curve only needs to change `lambda` at the top of the script and re-render, which is the point of parameterising the decay speed rather than hard-coding it: the diagnostic signature is a property of the mechanism, not of any single simulation run.
::: {.callout-tip title="What the simulation shows, in two numbers"}
Two ratios carry the argument of this post. First, the early-read effect exceeds the late-read effect, because the short-run window is measuring the steady state plus an undecayed novelty term. Second, the bottom-up sum of individual experiment effects exceeds the holdout-based cumulative estimate, in the same direction as the roughly 2x naive multiple and roughly 1.3x bias-corrected multiple that GrowthBook (2025) report for the Airbnb case, and for the same reasons: selection-conditioned individual estimates and interactions across features that a bottom-up sum cannot see.
:::
## A diagnostic checklist: five questions before you trust a short-run result
You have a short-run lift on your dashboard, a launch meeting in an hour, and just enough uncertainty to make the decision uncomfortable. The simulation in the last section gave you a way to quantify the gap between reported impact and holdout impact, but you are not going to run that pipeline every Tuesday morning. What you need first is a fast diagnostic: five questions that tell you whether the number in front of you is likely to survive contact with the long run.
### 1. What is the typical novelty decay window for your product?
As Section II's diagnostic signal showed, novelty is not governed by calendar time alone. It is governed by exposure frequency. A feature that users encounter every day can burn through its novelty effect quickly, while a surface users see only occasionally may still be in adjustment weeks after launch. The practical question is not whether novelty can exist. It is whether you can name the relevant window for this product in days or weeks. If your readout arrives before users have completed a full cycle of typical exposure, the measured lift may be a reaction to first contact rather than an estimate of the stable effect.
For a product leader, the test is simple. If the answer is "we usually need ten to fourteen days before this surface settles," you have a decision rule. If the answer is "novelty probably fades eventually," you do not. The first answer can be compared with the experiment duration. The second is just a way of admitting that the dashboard is being asked to do more work than the design supports.
### 2. Do you have an active holdout, or are you relying on the bottom-up sum of individual effects?
This is the single highest-impact question on the list. Every other question asks how much to trust a number. This one asks whether the number has ever been checked against reality. The portfolio-level Winner's Curse from Section III means that a roadmap built from shipped experiments is selected twice: first by statistical noise, then by the launch decision. Individual A/B tests may be unbiased before selection, but the subset of "wins" that gets shipped is not a neutral sample of all experiments run.
The Airbnb case discussed in Section III is the reason this matters strategically, not just statistically. A team can report a sequence of positive feature lifts and still overstate the aggregate impact of the roadmap once interactions, decay, and selection are allowed to accumulate. Without a universal holdout, the team has no direct way to know whether its reported wins reflect product value, statistical luck, or a mixture of both. If the answer here is "bottom-up only," the honest interpretation of every other answer on this checklist is provisional.
### 3. Does your product have equilibrium effects (marketplace, social, or recommendation dynamics)?
As Sections I and II described, equilibrium effects are different from novelty effects. They arise when the treatment changes the environment in which other users, suppliers, recommendations, or shared resources operate. In a marketplace, treatment users may consume inventory that control users would otherwise have accessed. In a social product, treated users may affect untreated neighbours. In a recommendation system, treatment-induced behaviour may feed back into model training. Kohavi, Tang and Xu (2020, Ch. 22) discuss these as cases where interference violates the standard assumption that each unit's outcome depends only on its own assignment.
A positive answer here means a longer holdout alone will not fix the design. If the effect operates at the level of a market, graph, geography, or shared model, a user-level treatment effect curve can remain biased no matter how long the experiment runs. The question is not whether the experiment ran long enough. The question is whether the randomisation unit matches the level at which the causal effect operates. If it does not, you need a system-level design, cluster randomisation, geo randomisation, switchback assignment, or another isolation strategy suited to the interference mechanism.
### 4. When did you last reconcile a bottom-up estimate against an actual holdout readout?
This is the calibration question. Section VI made the cost of holdouts explicit, and Section VII showed how the reconciliation can be simulated end to end. The operational point is narrower: a team that has never compared its bottom-up estimate with a live holdout readout has no empirical basis for trusting either the size or the direction of its typical overestimation. It cannot know whether its roadmap accounting is mildly optimistic, badly inflated, or occasionally conservative.
Martini (2023) frames the value of holdouts in exactly this comparative way: the bottom-up sum of winning experiment impacts can look implausibly large, and a holdout gives a more accurate estimate of cumulative impact by comparing users exposed to the winning variants with users held back on the corresponding controls. The takeaway is not that one holdout readout becomes a universal correction factor forever. The value comes from periodic reconciliation. Over time, that comparison tells the organisation whether its experimentation program is calibrated, whether the gap is stable, and whether changes in power, launch thresholds, metric definitions, or product interactions are making the bias worse.
### 5. For this specific feature, is the risk novelty or primacy?
This is the question to ask before killing or shipping a single test result, not just before trusting an aggregate number. Section II's asymmetry argument is the key: novelty is visible and embarrassing; primacy is quiet and often unmeasured. A novelty effect makes the early lift look too good because users react to something new. A primacy effect can make the early lift look too weak because users are still habituated to the old experience or have not yet learned the new one.
The decision implication is asymmetric as well. If the feature is a visible UI change, a badge, a notification, or an attention-grabbing surface, a positive short-run result may be inflated. If the feature requires learning, discovery, or repeated use, a flat short-run result may be too pessimistic. The first failure mode ships features whose effect decays. The second kills features whose effect needs time to materialise. Treating both cases as generic "long-term uncertainty" is too vague to be useful. Before acting, name the likely direction of the bias.
No single question on this list replaces a holdout. The checklist is a triage tool for deciding where a universal holdout or feature-level holdout is worth the cost discussed in Section VI, not a substitute for running one. A team that answers these questions clearly has diagnosed its measurement risk; it has not made that risk disappear.
The whole argument of this post reduces to one uncomfortable point for anyone running an experimentation program at scale. Individual A/B tests are not the problem, and they are not what this post has been arguing against. The problem is that they answer a specific short-run causal question and the business reports them as if they had answered a different one. Novelty inflates that gap. Primacy suppresses it. Equilibrium reallocates it. And the very rule you use to select which experiments to ship guarantees the sum of your wins overstates what the sum actually was. Universal holdouts do not fix any of those mechanisms one by one. They replace the stack of local, per-test claims about lift with one direct measurement of what the roadmap actually moved. That substitution is what decides whether experimentation gets treated as measurement or as marketing at the next business review. Every experimentation program is running one or the other. Most are not aware they are choosing.
::: {.callout-tip title="The five-question checklist"}
- What is the typical novelty decay window for your product, in days or weeks?
- Do you have an active holdout, or are you relying on the bottom-up sum of individual effects?
- Does your product have equilibrium effects, such as marketplace, social, or recommendation dynamics?
- When did you last reconcile a bottom-up estimate against an actual holdout readout?
- For this specific feature, is the risk novelty or primacy?
:::
## References
Chamandy, N. (2016). Experimentation in a ridesharing marketplace. *Lyft Engineering*. <https://eng.lyft.com/experimentation-in-a-ridesharing-marketplace-b39db027a66e>
GrowthBook. (2025). Holdouts in GrowthBook: the gold standard for measuring cumulative impact. *GrowthBook Blog*. <https://www.growthbook.io/blog/holdouts-in-growthbook>
Hohnhold, H., O'Brien, D., & Tang, D. (2015). Focusing on the long-term: it's good for users and business. *Proceedings of the 21st ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD '15)*. <https://doi.org/10.1145/2783258.2788583>
Kohavi, R., Tang, D., & Xu, Y. (2020). *Trustworthy online controlled experiments: a practical guide to A/B testing*. Cambridge University Press.
Madhusudanan, V. (2022). Don't be a holdout holdout. *Statsig Blog*. <https://www.statsig.com/blog/getting-in-on-holdouts>
Martini, G. (2023). Holdouts: measuring experiment impact accurately. *Eppo Blog*. <https://www.geteppo.com/blog/holdouts-measuring-experiment-impact-accurately>
Shao, S., & Burke, C. (2022). Understanding the collective impact of experiments. *Etsy Code as Craft*. <https://www.etsy.com/codeascraft/understanding-the-collective-impact-of-experiments>
Shen, M., & Lee, M. (2017). Selection bias in online experimentation. *The Airbnb Tech Blog*. <https://medium.com/airbnb-engineering/selection-bias-in-online-experimentation-c3d67795cceb>
Uber XP Team. (2018). Under the hood of Uber's Experimentation Platform. *Uber Engineering Blog*. <https://www.uber.com/us/en/blog/xp/>
